A study of polysemy judgements and inter-annotator agreement

This paper describes two experiments on polysemy judgement and sense annotation. The first experiment enabled us to select the most polysemous words which were used in the second experiment, and which serve as test words for the evaluation of WSD systems. We show that this selection method yields results different from selecting words on the basis of their number of senses in a dictionary, and is more appropriate in the context. Both experiments show considerable disagreement among the human judges. Disagreement on sense annotation is particularly concerning, since it sheds some doubt on the very possibility of evaluating WSD systems. However, we show that a lot of the disagreement is due to the too fine granularity of sense definitions in dictionaries. We also show that clustering techniques can enable us to re-code the individual annotations according to a small set of "super-tags", and obtain a quite satisfactory level of inter-annotator agreement. Surprisingly enough, the natural clustering provided by lexicographers in the hierarchy of dictionary entries does not provide a substantial disagreement reduction, whereas "blind" data-oriented techniques produce satisfactory results. Beyond its practical goals, this paper therefore probably raises some more general questions about lexicographic practice and the adequacy of dictionaries to NLP tasks.